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ABSTRACT , | 

Since problems associated with the ^statistical 
methodology of educational research are becoming incre'asingly 
important, this paper examines a subset of problems associated with 
the analysi^^ and interpretation of aggregated data. Two major 
questions arise; (1) if a researcher knows' the level (e.g.^ 
individual^ teacher/classroom, sch^ool, school district) at which 
inferences are desired, what complications arise from analyzing data 
at different^ levels? and (2) are there general guidelines for 
determining the appropriate units of analysis in a given research 
context? Five research jcontexts in 'which grou^p observations can be 
used to estimate relationships among measurement of individuals are 
examined including contexts with missing observations^ fallibly 
measured . variables^ the economy of analysis^ anonymously collected 
information^ and ecological inference. In choos^:^ng units of analysis, 
appropriateness is a function of the questions asked and of the 
sampling and/or experimental unit. The former is\reflected in the 
conceptualization of the research objective while the latter can 
indicate the presence of statistical constraiijts on the level of 
i nference > Examples of issues and pr<5blems that arise wi.th each 
"concern are provided. (Author/DE) • 
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^rTl"DSrorN°E7Es^R°:v''«'^;.r data aggregation in feDUCATIONAL RESEARCH: APPLICATIONS* 

SENT OFFICIAL NATIONAL INSTITUTE OF 

EDUC'aTION POSITION OR POLICY . ^ 

. Lei*gh Burstein 

University of Wisconsin-Milwaukee 

In the last ten to fifteen years, the character and focus of edu- 

QQ .cational research has changed. A key indicator of the present trend 

rQ:^^ is the greater frequency of large-scale educational surveys and inves- 

1 ' tigations (Project Talent, Coleman Survey, National Assessment of 

V« — \ 

CD Educational Progress) . A primary impetus for this transformation is 

UJ 

the expanding propensity of the representatives of the various social 
^ ' science disciplines (particularly sociology and economics) to attempt 

to detail the complex phenomenon called "schooling". The effects of 
schooling are what Coleman and his colleagues (1966) sought to eluci- 
date, and what their critics and supporters (e.g.. Mood, £t al (1970), 
and Hosteller and Moynihan (1972)) attempted to clarify, N^deny, or deify. 
The topic was worth three volumes (plus massive technical appendices) . 
^. J ^^ayeske and his colleagues (1972), eleven volumes^by International 

Association for the Evaluation of Educational Achievement (lEA) (1^67,' 

1973, 1975-) , and a catalogue of studies and summaries by Averch £t al 
2 

(1972). The attempts to evaluate the major educational innovations of 
the War on Poverty also adopted the mode of analyzing school effects 
data that are massive both in terms of persons and characteristics mea- 
sured. 

Unfortunately, one does not expan^i from investigations of a few 

/ 

classrooms to nation-wide or even cross-national studies of schooling 
without encountering new and perhaps novel complications. And, if it 
is difficult to adequately control even a single-class, short-term experi- 
ment (and the evidence' indicates that it is), how can we hope to maintain 



*Paper presented at the Annual Meeting of the American Educati(^n«il Itrjsearch 
Association, April 3, 1975, Washington, D.C. 



Eg^c 00003 



control K for 500 cla.s.srooms or 200 schools or 10 countries? The answer 
is that we simply cannot, but it Is believed that the broader perspective 
afCorc'ed by the more '^macrocosmic camera" yields more realistic, better 
generalizable image o£ the phenomenon of schooling. 

Moreover, large-scale investigations of schooling enter the realm 
of socio-politics and generally become the instrument for policy analy- 
sis. This, too, is a double-edge sword — the deathknell of abstract 
theory and the awakening of socio-politico-economic consequences. Thus 

today we fin^ ourselves as much in-need of developing our political 

^ / * ^ - 

skills as we do our research capabilities. Given present political con- 
tingencies, there is a greater burden on the researcher to ensure that 
his analytical procedures and data interpretation are conducted in a 
manner that can withstand substantive professional critic^ism. 

Despite the introductory remarks, this presentation is not about 
the philosophy or politics of education per se. My primary . concern is 
with a subset of the problems associated with an increasingly important 
aspect o£ the methodology of educational research — the analysis and 
interpretation of aggregated data. Aggregated data are encountered in 
almost all large-scale educational studies pimply because schools are 
a^^gregates of their teachers and pupils, a^d classrooms are aggregates 

!f the processes and persons within. The grouping of data can be simply 
modest attempt to pare research costs and/or "scrub*' dirty data, and 

I ■ / 

fin these instarrces, at^gregation has Relatively innocuous consequences. 



The use of aggregated data can enhance or obfuscate efforts to identify 
the relations among measures of human behavior. Often, the social and 
political context of the investigation will determine whether data aggre- 
gation occurs and v/hether ii\terpretatlons based on aggregated data are 
enlightening or illusionary. 
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In an earlier document (Burstein, 1974b), three layers of problems 
related to aggregating data in educational research were .-identified : 

(a) problems in grouping of observations or change in units 
\ ' ' of analysis; 

(b) problems in cross-level inference, which is better known 

as the identification and analysis of contextual effects; 

and' • * 

(c) problems in determining the appropriate units of analysis. 
Here we shall focus on the evidence from research on "change in units of 
analysis" problems and the overriding issue of appropriate units of ana- 
lysis. The issues related to cross-level inferences have been considere 
in the Division G round table on "Contextual Effects" and v;e will not at- 
tempt to elaborate on that discussion, 

\ 

DEFINING TOE ISSUES 
Problems of data aggregation have important implications for edu- 
cational researchers who are interested in relations among observations 
on individuals. For instance, the investigator ©ay want to know the co- 
efficient from the , regression of student achievement on other student. 
characteristics. However, these measurements cannot always be examined 
at -the individual level. The data may not be obtainable or identifiable 
for e^^ch person, because of intact school or classroom reportinc^, or the 
school or classroom may be the sampling unit, or it may be too costly 
to analyze data at the individual level. Faced with such problems, ob- 
sarvatiohs on individuals are grouped according to, say, classrooms 
(schools) and between-group (e.g., classroom, school) regression coeffi- 
cients are. calculated . The investigator then may attempt' to make infer- 
ences about the relations among individuals from the results of the ana- 
lyses at the group level. 



\ 

' ' \ 

Given a situation like the one described above, two major questions 

arise: 

(1) If we know the level (e.g., 'Individual, teacher/classroom, 
school, school aistrict) at which inferences are desired, 
what complications arise from analyzing data at different 
levels?. -("Change in the units of analysis" or change-in- 
units problems.) ' /; • 

(2) Are there general guidelines .for determining the appropri- 
ate units of analysis in a givep research context? (*'Ap- 
propriate units of analysis" or appropriate-units problems.) 

Much is already known about change-in-units problems (Burstein, 1974 

I 

1975; Hannan and Burstein, 1974). The latter question subsumes chances-in 
units problems. In p;eneral,' however, the issues surrounding appropriate 
units in the social sciences are presently conceptuair'^and^are dealt with 
substantively on a case-by-case basis. He offer guidelines below that 
will hopefully clear up the blatant errors in selecting appropriate units 
and suggest vayr, oi proceeding when the appropriate choice of units is 
not obvious. 

CHANGE IN THE UNITS OF ANALYSIS ~ THE GROUPING OF OBSERVATIONS 

In general, complications can arise in translating relations from 
one level of analysis to another. Our primary concern is with change- 
in-units problems where the relations at the level of individuals are 
of interest, but the data are aggregated over individuals. 

The degree of investigator control over the aggregation of data is 
a primary determinant of the complications due to grouping. In certain 
contexts, group membership is determined in some natural way, e.g., school 
attended, or census tract, and is thus beyond the investigator's control 
except for exclusion of sampling units and individuals (limited or no 

4 nHjyooos 



investigator control). In other contexts, the investigator can manipulate 
the formation of groups, either completely or partially, • There are t^en- 
erally more options in the latter contexts for improving estimation. 
Research Contexts 

We can identify five research contexts ^Ln which group observations 
^ are used in estimating relations among measurements on individuals. These 
contexts include problems with (A) missing observations: (B) fallibly 
measured variables; (C')-^the economy of analysis; (D) anonymously collected 
information; and (E) ecological inference. The. degree of investigator 
control over the formation of groups varies according to context, Ther^. 
are also differences among contexts in the reasons why the methods of 
data' aggregation are used, how such methods are applied, and where they 
are principally applied. Table 1,. reproduced from Burstein (i97Aa) , sum- 
marifzes the characteristics of each context. 



Insert Table 1 here 



Complete Investi.crator Control , In the first three contexts, the inves- 



\ 



tigator has considerable flexibility about the choice of grouping; methods, 
However, the problems addressed in context (A) (Kl^-ne et^ al , (1971) and 
(B) (Blalock et al, (1970) have seldom been subjected to aggregation pro-, 
cedures as other statistical methods are considered more suitable, (See 
Affifi and Elashoff (1966, 1967) on the missing observations problem: 
and Madansky (1959), Blalock, et al (1970), Blalock (1971), and Wiley 
and Wiley (1971) on the measurement error problem,) The procedures 
for selecting a suitable aggregation procedure, are the same as in Con- 
text (C) and will be discussed along with that' context . 

Sound principles have already developed and demonstrated for dntn 
aggregation where the sixe and economy of analysis (Context C) is the 
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concern (I'rais & Aitchinson. (105/.): Cramer, (1964); Feige and Watts, 
(19/2): Hannan and Biirstein, (197/.); Bursteini (197Aa) . Compression 
oE data is an-^issue when a large amount of data is collected and it 
can be reasonably concluded that a substantial savings in research 
costs can be obtained with minimal information loss. As data sets 
such as the 1,000,000 studentL.Co.leman Survey (Coleman, et al , (1966)) 
and the 300,000 case lEA Six-Subject Survey (e.g.. Comber & Keeves, " 
■ (1973)) becoming increasingly common in educational research, the in- 
vestigator must clearly. weigh the merits of data aggregation as opposed 
to initial restrictions on sampling. 

Sampling limits data collection and cuts back on costs from the 
start, but leaves little possibility for extension beyond what is col- 
lected. Aggregation approaches require large collections of data 
which can be more costly if the information did not have to be collected 
in the first place. ^ However, the reduction in analysis costs through 
data aggregation can be substantial, and those aspects of the research 
that require it can still be conducted at the individual level. Thus, 
a "correct" choice is not always available and one mu.^t decide what 
considerations are most important. 

Data aggregation procedures can also be emploved in Ithe analysis 

I 

and reporting of confidential informa. 'on. When researchers analyze 
Census data reported by classifications such as "years of education" and 
"ethnicity", they are, in effect', examining relations involving aggre- 
gate measures. In studies like those with Census da ta J • information on . 
individuals is available and perhaps personally identifiable. However, 
practice and/or statutory considerations dictate that reporting.be done 
on some aggregate basis. Thus, confidentiality of information on individ- 
uals is protected by the reporting process. 



— .--^^he procedures for identifying accurate estimates of individ- 
ual parameters from data aggregated to maintain confidentiality 
are the same as in Context (C) . As long as individual-level information 
is collected in an identifiable manner, the investigator can use his 
knowledge of the individual-level relations of primary variables to the 
method of grouping to identify a procedure that will be particularly 
suitable for minimizing information loss through grouping. Our examples 
in Appendix A demonstrate how j^uidelines based^ on the "structural equa- 
tions" approach (Hannan (1971); Hannan and Burstein (1974); Burstein 
(1974a)) and statistics developed by Feige and (7atts (1972) can be used 

to identify grouping methods with minimal information loss in this con- 

\ 

text, \ . 

Anonymously Collected Information , The use of aggregation techniques 
for analyzing anonymously collected information (Context (D)) is a rela- 
tively new notion,^ What distinguishes this' context from the others is 
that it is impossible to match observations on all primary variables 
at the individual level because information on certain primary variables 
has been collected anonymously. An application for grouping in this 
conS^^l would be in a study of the relations between student achievement 
aSd student attitudes where attitude data has been collected anonymously. 

In order to use grouping methods in Context (D), the investigator 
collectrf^Hnformation on potentially suitable grouping c^haracteristics 
in addition to variables of primarly interest. The individual observa- 
tions are then collapsed into different groups and the parameters of 
interest can be estimated from the between-group relations. This pro- 
cedure is viable as long as the potential grouping characteristics are 
measured simultaneously with each primary variable rep,ardless of whether 
the information on the primary variable has been collected anonymously 
or with the individual identified. The grouping characteristics must 
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also satisfy certain conditions* necessqjry for precise estimation for 
all chanr.c-in-uni ts problems. In Appendix B we present examples where 
^ Individual --level rejations are estimated from aggres^ation procedures 
that can be employed when sojn^ data have been collected anonymously. 

We recognize that conducting research on confidential data pre- ' . 
sents very complicated social and political problems. . (See Boruch, 
(1971a, 1971b, 1972a, lQ72b)). There is a definite need fo'r the pri- 
vacy and protection of subjects in social research, but in many cases, 
limits on individual cpnf ideneiality are necessary for arriving at a 
better understanding of the socio-culturall milieu. Educational re- 
search is no exception to this dilemma, pere are many sd-calied facts 
based on confidential data which would disappear with the more adequate 
qualifiers found in individual-level data. Robinson' s'^ (1950) paper pro- 
vides an excellent example cf an aggregation- induced misinterpretation.^ 

The procedures sugge^ed in this paper^ offer individuals assur^inces 
of their anonymity, while maintaining the possibility of carrying out 
research on topics that can further understanding of the complex inder- 
actions ampnq individuals and 'institutions » Our basic premise in siigeest- 
ing aggregation procedures in this context is that individuals can lie 
protected through analysis ^methods which allow examination of relations , 
among human characteristics without directly identifying the participat- 
ing individuals. 

Ecological Inference . The topic of ecological inference (Context (E)> 
has been extensively discussed in the sociological literature (See Robin- 
son, (1950); Menzel, (1950); Duncan and Davis, (1953) Goodman, (1953, 195Q) : 
Scheuch, (1966) includes a particularly cogent discussion of the problem). 
Earliei debates cencered around methods for overcoming the "ecological 
fallacy'* of using a^real data (e.g., data aggregated by census tract or 
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state) to estltpate relations amonp, characteristics of individuals. 
The dramatic change cited by Robinson (1950) ;in the size of the 
correlation between illiteracy and race as a function of 'the units 
of analysis (.P5 at the rej>ional level, at the s t:a'te->l ev«-l , -.20 
at the individual level) warn us nol to glibly assume that s^rbup ^data 
provide the same information as data on individuals. 

_ Educational researchers have seldom considered the potential 
"ecological" fallacy of inferring relations between properties of in- 
dividuals from eco.^ogical (between-group) correlatio^ns. . Any examina- 
tion of recent rer esfrch on the effects of schooling witi indicate 

that many invest igatiors perform between-group analyses, often without 

\ - . • 

even addressing the/ question of whether the relations estimated at 

; 

the group level are applicable at the individual level. Proper^ties 

/ 

of students have been aggregated to the classroom level (e.g., Walberg, 
(1969)), the school level (e.g., Burkhead, Fox and Holland (1967); 
Hanushek (1968); Ka tzman (1963)), the school district level (Kieslinj*, 
(1969, 1970); Bidwell and Kasarda (1975)), the college level\e.R., 
Rock, et al . (1970); Baird and Feister (1972)), and even the state le- 
vel (Walberg and Rasher (1974)) I 

Teacher charact^istic? are often aggregated to the school level 
even when information on^lndividual students is used in the^nalysis 
This was, the case in the Coleman Report (Coleman, £i al . (19^6)) and all 
of its re-analyses (See, e.^.. Levin (1970, Michelson (1970), and Smith 
(1972)), and it was also true with the IFA studies (Hu.sen ejt al^, (1967): 
Combe and Keeves (1973); Purves (1973); Thorndike '(1973); and others). 
Thus it is impos'sible in these studies to match indivicfual^ students with 
data on their own teachers. Under these circumstances, it is unlikely 
that school resources measures can account for a large proportion of 
the variation in individual student achievement since the variation-*ln** 
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achievement among students within the same^chool accounts for the most 
of between-stiident variation* 

In the face of the potential complications from data aggregation, 
the track record for input-output studies' of the effects of schooling 
15 not very good. In a report on school effectiveness research for the 
President*^ Commission on School Finance, Averch eit al (1972) reviewed 
the major input-output studies to date. Of 19 studies cited, the school 

was the unit of analysis 6 times; the school district, 5 times; and in 

/ 

' 7,vschool and teacher variables entered the ai\^lysis as school-rlevel 

I ' \ ' ' 

aggregates, while student variables were entered at the individual 

/ 

level. Only a study conducted by.Hanushek (1970) matched individual 
students with their own teacher, Hanushek's (1972) followup on his 
earlier wofk and a recently reported investigation by researchers at 
the Federal Reserve Bank Phfiladelphia (Summers and Wolfe (1974)) are,. 
*:he only other major studies with students and teachers matched. 

All three studies using matched student^teacher data were based 
on a single school district and thus are not necessarily comparable to 
the larger studies that use schools or school <ifstricts as the units of 
analysis. We nted more such studies involving multiple school districts 
before we can .adequately grasp how much misinformation has been generated 
by the between-school and between-school district analyses, 

the point is that if we agree with Averch et al (1972) that "the 
researcher would like to examine the relationship among school resources 
an individual student receives, his background, and the influences of 

^ -j 

his peers on one hand and his educational out;pome on the other (page 38) V 
then we need to use individual-level data or ^t least tte^ssured that 
the aggregation process does not distort the relations! among important 
variables At this point it appears to be impossible to avoid distortion 
-through aggregation in thi^ context. In fact, at present, we can not be 
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sure that the variables that appear at one level of analysis will neces- 
sarily appear at other levels. 

■ • \ 

Data from the lEA studies of the factors influencing educational 

achievement can be used to illustrate the extent of our present state of 
6 * 

ignorance. This study involved twenty-one countries, considering six 
subject areas (Science, Reading Comprehension, Literature, Civics Educa- 
tion, English as a Foreign Language, and French as a Foreign Language) 
at three age levels (basica'lly, 10 year-olds, 13 year-olds, and students 
\xv their pre-University year). Ovef 700 student, teacher, and school 

/ 

Ncharacteristics wei;e measured* 

Among the myriads of analyses and sub-investigations in the lEA 
studies were between-student and between-school regression analyses 
with achievement measures as outcome variables. The independent vari- 
ables included home background factors (plus sex and age) (Block 1) , 
type of school and program indices (Block 2), school and teacher vari- 
ables (Block 3), and other student measures (Block 4^) , entered in an 
ordered fashion. We focus here on the data^Jrpm the, .United States for 
all age levels in Science, Reading Comprehension and Literature. 

As stated earlier, the teacher measures were included as school- 
level aggregates in both levels of analytses. Thus we might expect that 
their contribution to the variance accounted for would increase relative 
to home background factors as the analysis^4hif ted from the student lav**l 
to the school level, simply because they are then on more "equal'' ground 
witK' the otiier independent variables. But that is not the case as Table 
2 clearly illustrates. 



Insert Table 2 



For all age levels in each subject area, the contribution of home background 
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Table 2. Percentage of variance accounted' for by home background 
measures (Block 1) and recent learning conditions (Block 
3) in the Between-Student and Between-School analyses in 
three subject areas ^Science, Reading Comprehension, and 
Literature), ai>^' three age levels (10 year olds, 13 year 
olds, and pre-Uhiversity year (1,7)), from the lEA study 
of educational achievement in the Uaited States. 



Pfevcentage of Variance Accounted for 




Home Background Factors 



Recen^t Learning Conditions 
(3)^ 



Subject 
Area 



Science 



Reading 
Compre- 
hension 



Literature 



Median 



Age . 
Level 


Between 
Student 


Between 
School 


Between 
S tuden't 


Between 
ocnoox 


10 


176 


^ 668 


088 


079 


13 


21;7 


'672 . 


048 


105 


17^ 


184 ■ 


435 


081 

i 


131 


10 • 


198 


629 


039 


062 


13^ 


221 


668 • 


032 


055 


17" 




526 . 


029 


000 


.13 


185 


549 


077 


082 


17" 


■163 


34^ 


050 


223 




185 


589 


049 


081 



^These data are taken from a preliminary version of An Empirical Study of Educ ation 
in Twenty One Countries: A Technical Repor t, by Gilbert F. Peaker, which is due 
to be published later this year. They are reproduced here with the permission of 
the International Association for the Evaluation of Educational Achievement (lEA) 
who sponsored the report. These figures should not be cited without checking 
with the original source. 

''Decimal points have been deleted for table clarity. For example, 176 should be 
read as "17.6% of the variance is accounted for". 

^The pomposiuion o£ Block l^which was labeled "Home Background Factors", varies 
according to the particular subject area and age level but generally includes an 
composite index of father's occupation, -mother 's and father's education, and the 
presepce of a dictionary in the home plus student sex and student age. ^ 

"^The composition of Block 3, labeled "Recent Learning Conditions", varies according 
to the oarticular subject area and age level. The percentages of variance accounted 
for by Block 3 in the table above reflects their contribution after home background 
factors,' (Block 1) and Indices of the type of school and type of program in which the 
student is enrolled (Block 2) have already been entered. 

®Tha actual description of the third population of students is "all students in their 
pre-University year" pre«5umably twelfth grade in the United States. 

f„._ ..;,--.-~„-^-^-^'.,>-i«HoH ^n the.L-i.Mn;ature study. -0001.5 



^Ten year olds wereTiot -rncloded in. t^^^^i^^^f "^^^ study. 
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factors increases dramatically accounting for more than half of the 
between-school variation in achievement (compared to roup.hly^2P% .of the 
between-student variation) while the contribution of recent learning 
conditions increases only alightly inmost cases. 

The fact is that students tend to group together in schools with 
other students from similar home ba5i'kgroun3s (Comber and Keeves, 1973). 
Therefore, aggregating data to the school level does not greatly reduce 
home background variation. As a result, home background factors become 
more influential, rather thnn less so, when the unit of analysis shifts 
— from--the--student to the school. It /iso means that it becomes harder - 
for school resources variables to significantly contribute to variation 
in achievement as the home background variables have already accounted* 
for a "larger chunk of the smaller/ pie" through their earlier entry. 

The picture is further complicated by the fact the the recent learn- 
ing conditions which appear to affect achievement variation in the TEA 
study are not necessarily the same in the between-student and between- 
school analyses. For example, the pupil/teacher ratio, the opportunity 
to observe experiments and the presence of a science teacher for 10 year- 
olds in Science, and the number of hours^ of homework per week reported 
by the student for 13 year-olds in the Science study make significant 
contributions in the between-student analysis but fail to appear in the 
between-school analyses. Examples of irtportant variables identified ^ 
in the between-school analyses that do not contribute in the between-\^ 
student analyses are, the use of audiovisual methods (10 year-olds in Sci- 
ence) and a composite measure of the student's perception of ^the schooi 
environment (1,3 -year-olds in Science). And on and on and on . . . , the 
same lack of consistency in every subject in virtually every country. 

If the lEA studies accurately reflect present knowledge, we obviously 
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have a lot left to learn about aggregation over students and teachers 
within schools and within larger administrative units. ,The guidelines 
we prescribe later on and the examples we present suggest that it is 
possible under certain conditions to find reasonable estimates of in- 
dividual-level relations from group data or to determine when grouped 
data yield highly biased estimates of individual-level parameters. 
So far however, the utility of these guidelines have been demonstrated 
only for the case where the characteristic that determines group mem- 
bership ( we call it the ^'grouping variable") is measured on at least 
an ordered scale (see Hannan and B.urstein, (1974), Burstein (1974a)), 
which i8 not the^^gase here. 

/ 

We have some notions about how to proceed when nominal ^'ecological" 
variables such as school and school districts are used to form groups, 
but have been unable to obtain the necessary data to try out our ideas 
on any reasonable scale. The ideal data set for such an investigation 
contains all variables measurable at their lowest possible level — 
individual students matched with their own teachers and characteristics 
of their own school setting for students from multiple schools and, 
hopefully, multiple school districts. Then we would do the following: 

1. Individual Model • 

a. Determine the best model for school effects at the 
individual level; 

b. Agf^regate inputs from the model (independent variables) 
successively to the levei of classroom, school, and 
school district; 

c. Aggregate both outcomes aftd inputs to classroom, 
school, and school district levels; and, 

7 

d. Attempt to identify the factors that coincide with 
changes in coefficients across "levels. 

00017 



2, Classroom Level 

a. Find the best model for school effects at the classroom 
level; 

b. If different from the model at the individual level, 
aggregate and/or disaggregate inputs to the level of 
the school, school district, and the student; 

c. Aggregate and/or disaggregate both outcomes and inputs 
\^ to the level of the sichool, the school district, and 

the student, ^• 

3. Repeat //I and //2 .starting with the best model at t(ie school 
level. 



4, Repea^t ,//l, //2, and //3 starting with the best model at the 

school district level, 
\^at we would expect to find is that (i) there are substantial dif- 
ferences in tlie magnitudes, of the coefficients across levels for a given 
model, (ii) different variables enter the models at the different levels, 
and (iii) the coef f iciejits for variables that appear in multiple *'best** 
models will differ across models even at the same level of aggregation, 
The^ only conditions that would lead us to expect results other than the 
above would be if schools (classrooms, school districts) were random 
groupings of students and teachers, or if there were no between-school 
(classroom, school district) variation in performance on outcome measures 
that could 'not be accounted for by input variables in the "best" models. 
Neither of these conditions is likely to occur. 

The id^cil study described above has not been done and probably 
cannot be done with any presently available sets of educational data. 
The closest thing to it so far is the Hannan, Freeman, and Meyer analysis 
(1975) using the Merder data which Ucks the match between students and 



00018 



16 

their teachers. They found the expected changes in the models across 
levels and large differences in the coefficients for the various input 
variables (Hannan discusses their work further in his own presentation 
(Hannan (1975)), I 

CHOOSING THE APPROPRIATE UNIT OF ANALYSES 
In choosing a unit of analysis, the researcher is in effect making 
some choice with respect to th<j level of aggregation which suits his 
purposes. Whereas change-in-units are limited to selecting among alter- 
native possibilities for compressing data or interpreting the information 
loss where the choice of grouping method is beyond the investigator's con- 
trol, choosing the units of 'analysis involves the specification of a set 
research foci in addition to determining an aggregation-disaggregation 
scheme^ In general, appropriateness is a -function of (a) the question 
asked and (b) the sampling and/or experimental unit. The former is 
-reflected in the conceptuailization of the research objective while the 



latter can indicate the presence of statistical constraints on the level 
of inference. Below we discuss and provide examples of the kinds of is- 
sues that arise with each concern, . • 
Conceptual Unit of Interest 

The issue Kere is the question of what is the research objective. 
If observations are generated from an appropriate sam^ing design, the 
data at every level in the ideal study — pupil, classi^oom (teacher), 
school, school district and so on — can be used to Investigate certain 
empirically based questions. Though aggregate data are usually inappro- 
priate for studying properties of individual members of the aggregates, 
the investigator may be interested in the behavior of the aggregate units 
themselves. 
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Classrooms as Units . Studies of group process in educational settings 
can be meaningfully conducted at the level of the classroom. For ex- 
ample, though the methodology is not entirely satisfactory, Walberg 
(1969) appropriately concentrates on the classrooms as the unit of 
a^nalysis in studying classroom climate. Walberg *s purpose is to "re- 
plicate the work on the effects of classroom climate on learning and. 
to investigate . . . effects of student biographical characteristics 
. • . on learning for the class as a whole (Walberg (1969), p. 529, 
emphasis added)". Apparently, he has no interest in applying his find- 
ings to individuals. 

Schools as Units . There are many situations in which the school can be 
the appropriate unit of analysis, though most of the studies cited in 
Averch et al^ (1972) do so incorrectly. If the purpose of the investiga- 
tion is to identify unusually "effective or ineffective" schools or to 
depict {differences in practices among schools, between-school analyses 
are called for. A Rand sponsored study by Klitfjaard and Hall (1973) 
contains a particularly thorough treatment of the process of identifying 
effective schools. 

,We (Burstein, Kremer, and Gemoll,' in progress) are currently ana- 
lyzing^3 years of school-level data which iA^cludes achievement, student 
background, school resources, and teacher experience indices. Cur pur- 
pose is^.to identify those schools which appear to be unusually effec- 
tive or ineffective in terms of proportion of low scoring and high scor- 
ing students .after controlling for student background characteristifcs 



(ethnicity\ mobility, etc.). Once such schools are identified, the 



\ 
\ 
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.school resource and teacher training data will be examined and further 
information will be gathered in a case-study fashion on each effective 
(ineffective) school. 
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The exercise described above might be conducted by the research 
and planning staff of a system in order to determine if there are cost 
effective ways to allocate the limited discretionary portion of school 
funds. This type of study is also coct effective in that the research 
staff expends mosc of its energy intensively investigating that subset 
of the school system that offers the highest potential payoff. 

An unusually enlightening (from the point of view of appropriate 
units) series of studies has been conducte'd by research psychologists 
at ETS on various aspects of college environments. The college was 
the primary unit of analysis in a study by Rock et al (1970) of the 
•-relationship between college characteristics and student achievement; 
by Rock et al (1972) of the interaction of college effects and student . 
aptitude (we return to this article later on),, and by Baird and Feister 
(1972) of grading standards. ' • 

The Baird-Feister study exemplifies the perspective of the whole 
series. The overriding question in Baird and Feister's investigation 
■ was whether college grading standards are affected by changes in the 
abilities of entering students. As part of the work they sought to 
determine whether the same grade reflected the same level of student 
performance from college to college and from year to year in. the same 
college. They consistently and appropriately examined college-level 
behavior in order to answer their questions. 

School Districts as Units . For the most part, the conditions under 
which one might contemplate treating the school district as the unit of 
analysis parallel those of the school as unit. _Jhe research division 
of a state department of education might try to identify the characteris 
tics of unusually effective or .ineffective school districts in the same 



" • ■ . V/ ' 

fashion as we are doing with schools. There is again a potentially sub- 
stantial savings in research costs and perhaps the results can provide 
guidelines for cost-beneficial targeting of* state funds. 

Another instance in which the school district is the desirable 
unit is in studies of the administrative intensity and the like. A ' 
recent study by Hannan and Freeman (1975) demonstrates this usage of 
between-distric t data-. 

We could carry this line of inquiry on to higher levels, such as 
states and countries, and always be able to identify som-^ H^**-stidn where 
the appropriate unit is a specific collective property. The mapping 
of appropriate units to answer specific questions is invariably logical. 
It is only in using a unit that is illogical at face value where problems 
generally arise. 

For example, though their conclusions are appealing^ it is the height 
of folly to believe that Walberg and Rasher (1974) have avoided the method- 
ological shortcomings of the Coleman study by using state-level, data , es- 
pecially with 1969 and 1970 selective service examination failures ^s the 
outcome variablesi Walberg had previously cited (if not read) Robinson* s 
(1950) paper (Walberg, 1969, p. 530) and should know better than to treat 
states as random groupings of persons. Besides- anyone who was eligible 
for the draft in 1969-70 can tell you that (a) women did not take military 
service mental tests and (b) the years 1969-70 were not the times to try 
to pass the test (a case of the winners are the. losers, in this writer^s 
opinion) . - ^ ^ 

Statistical Considerations 

The important statistical considerations are of two types — those 
related to sampling and experimentation and those related to model spe- 
cification. These problems receive attention in two entirely different 
literatures, the former being of concern primarily to statisticians and 

.jma 
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psychologists, and the latter to econometricians and sociological "mo- 
delers*** 

SamplinR and the Experimental Unit . The problems with sampling and 
the experimental unit represent familiar terrain. In experiments where 
sampling units are groups of individuals (e»g*, class rooms) , between— 
group analyses must logically be conducted even when the relations among 
measurements on individuals is of primary concern* The investigator 
lacks control over group membership and is therefore unable to deter- 
mine how the required grouping procedure affects variation and covaria- 
tion of the study variables. Under these conditions, the possibility 
of inferring relations at the individual level is limited. 

In any case, the sampling' of groups can present a particularly' 
complex type of aggregation problem, since questions regarding sampling 
bias arise in addition to concerns about level of inference. For in- 
stance, the investigator needs to know whether the sampled classrooms 
are representative of classrooms in the universe to which he wishes 
to gGiwralize in order for the between-group analys(!S co make sense. 

Another facet of the sampling problem is the question of experi- 
mental unit. The ex:perimental unit is "the smallest . , . collection 
of experimental subjects that have been randomly assigned to different 
conditions . . ^ and have responded independently of each other for the 
duration of the experiment (Glass and Stanley (L972), p. 506, emphasis 

' . ; 

added)*'. Thus, according to Glass and Stanley, studies involving intact 
classrooms should legitimately analyze classroom means. (Lewy (1972) 
states basicfally the same point.) 

Glass and Stanley's arguments are sobering. In the extreme, they 
mean that between-student analyses are never appropriate with data pro- 
vided from classrooms even if students were randomly assigned and sampled. 
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Our only salvation would be the nuances of random grouping and its ef- 
fect on the relation of between-group to between-student analyses (see 
earlier discussion)/ Since most studies of the effects of schooling 
are at best quasi-^experimentd, the question of experimental unit is ' 
just one more methodological hurdle. 

Specification Bias. We will not go into detail here as there is already 

substantial literature on the topic of specification • bias and its 

I 

interrelation to aggregation bias (Hannan (1971); Feige and Watts (1972) 
Hannan and Burstein (1974); Hanusheki et al (1974); Burstein (1975)). 

The argument goes like this. The inability of social scientists 
to base their analyses of statistical models of human behavior on a well 
defined theory often leads to misspecif ication through the inclusion of 
redundant (collinear) regressors or, more importantly, through the exclu 
sion of causally relevant measures. Model misspecification as described 
above affects the relationship of a sample estimate of a coefficient to 
its population value. 

Aggregation bias is a form of specification bias in that it arises 
through a lack of independence between the variables in the model and 
their disturbances terms. Furthermore, data aggregation can only exacer- 
bate the problems caused by other forms of specification bias (Hannan 
and Burstein (197A); Hanushek, et al (1974)). / 

c 

Investigations of contextual effects (e.g., the effects of school 
environment on performance) have been particula^i^ly prone to the problems 
of model misspecification. Hauser (1970, 1971, 1972a, 1974) has demon- 
strated that so-called context effects virtually disappear once all 
relevant indiyidual-level variables have been included in the model. 
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Appropriate Cross-Level and Sub-Level Designs 

There are situations which call for mul^tilevel analyses of data* 
We cite below three studies in this vein. 

Group Anchored Verses Sample Anchored Measures . Lewy (1972) suggests 
that the appropriate type of correlation coefficient is a function of 
both questions one asks and the type of variables one examines. He 
points out that if measures are "group-anchored" (relative referents 
such as teacher grades^ and student's self-appraisal) as opjposed to 
"sample-anchored" (absolute referent such as standardized achievement 
measures), then pooled within-group correlations, rather than total 
individual-level correlations, convey the right information. Further- 
more,- Lewy demonstrates how the relative magnitudes and signs of be- 
tween-group correlations, pooled within-group correlations and total 
individual correlations vary according to type of measure (group-an- 
chored or sample-anchored). The data he presents are provocative 
and suggest that multilevel analyses are necessary for school effects 
data. 

Within-College and Between-College Analyses . Rock et al (1972) attempted 
to find groups of colleges that are equally\ effective using within col- 
lege variation to estimate interactions between student aptitude and 
college effects. They (a) calculated within-college regression lines, 

(b) clustered colleges on the parameters of the within-college regressions, 

(c) generated discriminant functions to check for statistical distinctions 
among clusters of colleges on the basis of the within-college parameters, 
and (d) identified the descriptive measures of the colleges that success- 
fully discriminate among the clusters. Thus, Rock and his associates 
utilized both within-school and between-school analyses to achieve their 
objectives. 

? op02S> ;; * 
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Analysis of .Hierarchical Data ^ Kiesllng and Wiley (1973) argue that 
5 

school-le^'^,1 indices such as average daily attendance do not convey 
independent information for each pupil within the school and thus 
should not be included in between-student analyses: If there ar6 
only say, 40 schools, then there are 39 degrees of freedom for school, 
no matter how many students are involved* They advocate (a) performing 
within-school regressions, (b) aggregating the information to. the level 
of the school, and (c) entering the aggregated within-school model as 
a variable in a between-school analysis in order to minimize bias in 
estimating parameters at the school level. Kiesling and Wiley demon- 
strate their techniques with data from the Coleman study, and they in- 
deed imp'tove the estimation of the effects of school inputs in this fa- 
shion. 

The three studies discussed ^bove reflect new and, perhaps improved, 
directions for the analyses of effects of schooling. Each analysis is 
conditioned on the questions asked and the process by which variables 

are generated and can logically affect one another. They also demonstrate 

I 

once again the complexity of choosing the appropriate units and in doing 

m 

SO, aptly summarisje our conclusions. 
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FOOTNOTES 

s? 

This paper is an .outgrowth of research supported by the National 
Institute of Education (Grant No. NIE-C-74-0123 to Vasquez Associates, 
Ltd.). Lee J. Cronbach, Edward L. Feige» Michael T. Hanhan, Robert 
H. Hauser, Thomas' Re Knapp, Harry Lutjohan, Carlyle E. .Maw, Ingram 

. Olkin, and David E. Wiley have all made contributions to the ideas 
expressed herein through their comments and suggestions over the past 
3 years^^Mi-ch'^eT'Baenen, James Knoop, and Jan Sh^nahan assisted with 

' '*th*e "data ^'analysis for Appendices A and B arici Margie Mika prepared 
r.eaaable copies of the manuscript.^ The errors and misrepresentations 
that remain are solely the responsibility of the author. 

c 

2 

This list does not even begin to replresent the work on related topics 
such as the study of contextual effects. See, e.g., Hauser, 1972, 1974. 

3 

We are trying to distinguish between data such as provided by the U.S. 
Census and the data from the lEA survey. 

^See Feige and Watts. (1972) for an example involving Federal Reserve 
•data. / 

^Boruch (1971, 1972) mentions the procedure in his writings on conduct- 
ing research with confidential data. 

She author wishes to thank Toirsten Husen and Roy W. Phillipps for their 
' .permission to reproduce the data from the Technical Report of the lEA 
study. 
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APPENDIX A: Empirical examples of the estimation of standardized 
regression coef f ici^ents in the single regress'or case 
— A discussion of^Hannan-Burstein bias prediction, 
and Feige-Watts techniques. 



This appendix performs two functions. First, it provides two empirical 

examples that are more comprehensive than those published in Hannan and Bur- 

/ ^ 
stein (1974) and Burstein (1^7^a). All variables have been standardized 

so that we are estimating correlation coefficients as well as regression 



coefficients in this single regressor case. 



J regre 



r 



Second, the example^ also represent the first appearance in the educa- 
tional literature of tecl^tiiques developed by the economists Feige^and Watts 
(1972) Their techniques, provide a statistic for the discrepancy between 
grouped and ungrouped estimates of regression parameters. In the simple mo- 
dels presented here', we are able to consider utility of the Fei^e-Watts 
formulation relative to the prediction of bias from the Hannan-Burstein approach. 

Table A,l includes a short description of each grouping variable. The 
primary variables in the two examples are self appraisal of academic abilities 
(SRAA), a weighted composite of ten items asking the person to i^ate himself 
in various academic skill areas; total score from an achievement test battery 
given during college orientation (ACH) ; and total score on the Scholastic 
Aptitude Test (SAT) . In each example we wish to estimate from the regras- 

\ 

sion \ 
Y = e^yX + u . 

Tables A. 2 and A. 3 contain estimates of important parameters from the a 
structural equations for the standardized regressions of SRAA on ACH and' ACH 
and SAT and a grouping variable Z: (See Hannan and Burstein (1974); and Bur- 
stein (1974a) for 'details on the development of the approach.) 

Ungrouped: ^ " ^x- ^Z-X^' " 

X = B^^Z + V * 

Grouped: Y = ^yx-?^ '''YZ.X^' " 
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In Bur stein <197Aa)and in Che present discussion, we categorized the 
grouping variables (Z) according to the following procedure: 

I. Z directly .related to both Y and X ~ I&yz-x' ^^^^^^YZ-X^ ' '^Xz'^^^^^^XZ^ ' 

II. z, directly related to Y but not to X, ~ 
I !6,,.,-l>3SE(6,^;^);|6^,|<3SE(3^^). 

III. Z directly related to X but not to Y ~ 

IV. Z not directly related to either Y or X ~ / 
ieY7.vll3SE(BY2.x)HeJl3SE(8^2). 

Our expectation is that Category III grouping variables yield the best 
estimates (in terms of small bias and mean squared 'error) because they 
most nearly parallel grouping on the regressor (which is known [Prais and 
Aitchinson (1954> , Cramer (196A)] to yield the best estinates of all pos- 
sible grouping procedures.) Category IV grouping should yield estimates 
that are unbiased but are relatively inefficient. This occurs because group--^ 
Ing on a Category IV variable which forms m gro.ups yields essentially 
the same results as basing one's estimate on a random sample of m ob- 
servations drawn from the original N. (See Cramer (196A) and Feige and Watts 
(1972) for details). We have yet to see a Category II grouping variable but 
expect it to behave more i like Category IV grouping than any other cate-- 
gory. Category I variables should yield relatively poor estimates which 
tend to be inefficient as well as biased. 

Tables A.4and A. 5 contain the estimates from grouped data of the stan- 
dardized regression coefficients and their standard errors for the two 
models plus estimates of the predicted bias based on the formula presented 
in both Hannan-Burstein (197A, essentially equation (17) on page 386 for 
the standardized case) and Burstein (1974a, page 27, again modified slightly) 
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and at the bottom of each table Feige-Watts F values are also Included 
in the tables and will be discussed later on, A detailed discussion 
of these tables appears elsewhere (Bursteiri, (1975*)) but^a few ob- 
servations can be made, 

1. With only one exception (CLIMP in the regression of SRAA 
on ACH), Category III variables yield estimates ,with smaller 
bias thaa any other variables except TD2? which formed at 
least ten times as many groups, 
2.. The standard errors for ail estimates are relatively large 

which is not surprising given the limited number of groups 
2 

in most cases, 

3, The standard errors for Catep,o'ry III grouping are generally 
smaller than those from grouping on a Category* I variable 
which forms a comparable number of groups, ^ ^ 

4, The truly poor grouping methods (e,g,,ANTDEG and QCJO^^n the re- 
gression of SRAA on ACH) are clearly identified and their huge 
biases predicted with accuracy. The 8 (out of 17) variables, 
with largest predicted biases had the largest observed biases 
for th6 regression of SRAA on ACH; the 6 variables with the 
largest predicted and observed biases from the regression of 
ACH on SAT were also the same, 

5, The results of predicting the bias for Category III and IV 
grouping were mixed. There are sign differences between 



Grouping on the regiressor and regressand have been placed first in theit 
respective categories (III and I) to indicate their special significance. 

2 

The computer procedure used to find the hetween-group regressions 
weights each group mean by the number of observations in the group and 
thereby basec the standard errors of the. coefficients on the entire sample 
of 2676 observations. We have resca led the standard errors from the grouped 
data by multiplying by /(N-l/(m-l) to reflect the actual number of obser- 
vations on which each coefficient is based. 



predicted and observed bias^for some variables (PARING In the 
ACH-on-SAT regression, IDl in the SRAA-on-ACH (regression); 
some cases of underprediction by as* much as .1 (IDl in the 
ACH-on-SAT regression, CLIMP^ in the SRAA-on-ACH regression) ; 
and the predicted bias for PARING in the SRAA-.on-ACH regres- 
sion is ,1 largej than the observed. However, overall, 7 . 
out of 10 of the groupings from both models (5 from each) with 
smallest observed bias were among the 10 with smallest pre- 
dieted bias. Only 2 of the 14 variables with predicted bias 
less than |*l| had observed bias greater than .|.l| (IDl in the 
ACH-on-SAT regression and CLIMP in t^e SRAA-on-ACH regression) • 
The moderate success of our attempts to predict bias in these simple 
cases is encouraging, given t:he small number of groups formed by most vari- 
ables and problems such as nonlinearity of relations, skewness and other 
factors affecting the grouping. characteristics. By use of compositing 
techniques first described in Burstein (1974a) and discussed in Appendix 
B, we can improve the accuracy of our estimates of individual-level parameters 
Feige and Watts Technique 

Feige and Watts (1972) developed a measure of the divergence between 
grouped and ungrouped estimators, ^ and b, in the multivariate , case. They 
attributed this divergence to three sources ~ (i) specification bias, 
(ii) bias introduced by a grouping Chat is not independent of the distur- 
bances from the structural model, and (iii) sampling error induced by 
the loss of information in grouping. We summarize the Feige-Watts ana- 
lysis below. 

We are interested in estimating B from the model 
Y = XS + u , 



The least-squares estimators of the regression parameters from raw 
data and their variance-covariance matrix are given by. 
b = (X'X)""''"X'Y 

and 

V(b) = a^CX'X)""-^ . ' . ' 

. In order to generate ^grouped data, a m x N grouping matrix G is 
introduced which transforms .the raw data to a set of m rows. The ith 
rows of the transformed matrices contain the mean values of the vari- 
ables for the ith group; i.e., the matrix (Y, X) is replaced by 

(Y,X) = (GY,GX) . 
t - — . 

Let H be a N X N matrix which produces the same grouping as G, but 
replicates the mean rows to accomplish the weighting that is necessary 
for unequal group sizes. H is related to G by: 

H = 9' (GG')"'^G. 

Using the H transformation, the estimates of 6 and their covariance 
matrix f,rom grouped data can be written 

B = (X'HX)"'"^X*HY 
and 

V(B) = a^(X'HX)""^ 

The divergence of grouped and ungroup^d estimates of 6, 
A(H) = b - B , 

has a zero mean and variance-covariance matrix equal to 
cov[A(H)] = o^{(X'HX)"-^ - (X'?)"""^) . 

< Let e = Y - XB so that e'e is the sum of squared residuals from the 
between-groups regression. Assume additionally that the disturbances u 
are normally distributed. Then the quadratic forms ' 



2 

0 

u 

and 



are variables with k and m-k^ degrees of freedom, respectively. 
If the model. is correctly specified and H and u are independent: 

(Q,/k) 

F - . 

(Q^/k-k]) 

is distributed as An F statistic with k and m-k degrees of freedom. 
Valusa of F beyond the critical values of the F-distribution indicate 
differences between estimators that could not be attributed to sampl- 
ing error. Hence good grouping methods yield small F values. 

To illustrate their findings, Feige and Watts examined 20 regres- 
sion equations generated from income and dividend information pro- 
vided 5393 banks to the Federal Reserve System. The seven group- 
ing rules they used included a random procedure and geographic and 
financial asset indices. There were also 3 levels of aggregation — 
slight (3 observations per group), moderate (30 observations) and 
drastic (100 observations). Thus 21 grouping methods were possible 
for each equation although the article only discussed a few. 

Certain of the Feige-Watts equations were quite sensitive to the 
choice of grouping method. The reported F values ranged from .02 to 
84.96. All the F values were significant at the .05 level for one 
equation," while grouping produced no significant F tests for other 
equations. 

In every case, slight aggregation was superior toother levels • 



Thus, a large number of groups again proved to%e desirable. 

The models in our example are much less complex than the ones 

Feige and Watts considered/ Because^^ we have only a single regressor 

\ 

and- the variables are standardized, the .formula for Fleige-Watts measure 

of divergence simplifies to 

2 ^ * 

F- = A (a(A)) , . ^ • 

a (res)/m-l 
where A = B~ - b^x 

-2 

^(A) = r_i— 1 1 " = (N - 1) ( 1 " ^x) 

and (res) is residual sum of squares from the' aggregate regression* 
As can be seen from Tables A, 4 and A, 5, there are, in general, 
large F statistics for what we have called Category t grouping variables 
and small F statistics for Category III and IV grouping. Grouping on 
the regressand provides the largest F statistic and grouping on the re- 
gressor, the smallest. In fact, 11 of the 12 variables with J values 
less than 1,0 result in observed biases smaller than ,1, 

The Hannan-Bur stein bias predictions and the Feige-Watts statistics 
both show signs of promise for providing guidance in choosing the op- 
timal grouping methods. But, as our example demonstrates, the results 
from these procedures do not always conform to our expectations. Until 
we can identify only good grouping methods and eliminate all poor ones, 
we will have to continue to improve our understanding of the grouping 
process. 
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Table Information on grouping variables. 



Variable 
Identification 



ID2' 
IDl 
HSGPA2 

SAT2 

ACH2 

PARING 

REPGPA 

POPED 

SRAA2 

ANTHIDBG 

HSMATH 

HSPHYS 

NODOOK 

PARASP 



Description 



Last 2 digits of student identi- 
fication 

Last digit of* student identifi- 
cation 

High school's report of student's 
grade point averager on a 4-point 
scale (highest 2 digits) 

Highest 2 digits of Itotal score 
fron the Scholastic ^titude Test 

Highest 2 digits of Total score*. ^ 
fron the Achievarvent Battery 

Student's best estimate of 1970 
parental incctne before taxes 

Student's report of average grade 
in secondary school 

Student's report of highest level 
of formal education obtained by 
his father 

Highest digit and sign of corposite 
academic self-opinion 

Student's anticipated highest aca- 
demic degree 

Student's report of niinber of semes- ^ 
ters of high school mathematics 

Student's report of number of semes- 
ters of high school physical sciences 

Student's report of number of books 
in the home 

"What is the highest level of education 
that your parents hope you vdll corplete?" 



Number of Groups 
Af Aggregation 

100 



7 



/ 



10 
23 

13 
10 
10 



ERIC 



00035 



Table A.l(Continued). Information on grouping variables. 



10 



CLIMP 

OOLEFF 

QCJOB 



"My grades are markedly better in 
courses that I see I will need later." 

"I often wonder if four years of col- 
lege will really be worth the effort." 

"I often wish that I were offered a good 
job now so I wDuldn't have to spend fovir 
years in college." 



4 
4 
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Table A. 2 Estimates of parameter| relating ACH(X) and SRAA(Y) to possible 
{grouping variables (Z) 



VARIABLE 
NA>E 



GROUP SIZE 



6. 



PARAMETER ESTIMATES 



YZ'-X 



3 



XZ 



YZ 



^X 



ID2 


100 


.008 


.020 


'019 


.189 


IDl 


10 


-.011 


-.042 


> 

-.033 


.078 


" HSGPA2 


23 


.123 


.535 


.370 


.552 


SAT2 


13 


.406 


.827 


.566 


.831 


ACH2 


10 


,.070 


.983 


.522 


.984 


PARING 


10 


.028 


.070 


.064 


.122 


REP CPA 


7 


-. 258 


-.490 


-.455 


.510 


POPED 


.'6 


.073 


.139 


.145 


.150 


SRAA2 


5 


.819 


.476 


.885 


. 4'31 




5 


.186 


4 1!;6 


..264 


.159 


HSMATil 


5 


-.066 


,479 


.202 


.489 


HSPHYS 


. 5 


.045 


.318 


.209 


.365 


NOBOOK 


5 


.122 


,146 


.196 


.148 


PARASP 


5 


.133 


.066 


.186 


.077 


CLIMP ' 


4 


.003 


.147 


.074 


.163 


COLEFF 


4 


.121 


.134 


.189 


.144 


QCJOB 


4 


.145 


.105 


.199 


.113 


^All variables 


have been 


standardized 


prior to 


grouping so that 




^' = °x = ^z = 


^' ^XZ = 


^XZ' ^YZ 


= ^z- 
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Table A^3 Estimates of parameters relating SAT(X) and ACH(Y) to possible 
grouping variables (Z)^ 











PARAMETER 


ESTIMATES 




VARIABLE 

NAME 


GROUP SIZE 
(m) 


6 


YZ'X 


hi 


^YZ 




idtz 


100 




.014 


.oos' 


.020 


.186 


IDI 


10 




.003 


-.046 


-.042 


.069 


\ 

^ \ 

HSGEA2 . 


23 




.164 - 


.488 


.535 


.517 


SAT2 


13 




.042 


.987 X 


~828 


.980 


ACH2 . ' 


10 




.916 


.827 \ ;983 


.835 


PARING 


10 




.006 


.076 


.Q70 


."146 


REPGPA 


7 




.124 


-.468 


-.•490 


.498 


POPED 


6 




.007 


.157 


.139 


.169 


SRAA2 


5- 




.054 


.520 


.476 


.531 


ANTDEG 


5 




.039 


.140 


.108 


.141 


HSMATH 


5 




.214 


.346 


.480 


.349 
















HSPHYS 


5 


.109 


.257 


.318 


.294 


NOBOOK 


5 




.025 


.203 


.146 


.204 


PARASP 


5 




.007 


.087 


, .066 


.101 


a IMP. 


k 




.009 


.165 


.147 


.185 


COLEFF 






.039 


.114 


.134 


.134 


,QCJOB 


4 




.007 


.118 


.106 


• .123 



a 

All variables have been standardized prior to grouping so that 
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Table A. 4 Kstlnvite.s from r,rouped data of the st.nndarrfized coef<%iencs 
from the rcy*resslon of SRAA on ACH, and bias prediction using 
Hannan-Rurstein and Feige-Watts procedures • 



VARIABLE 
NAME 



NUMBER OF 
GROUPS B-/ 
m) 



SE(B--)' 



OBSERVED 
BIAS A 



HANNAN- 
BURSTEIN 
PREDICTED 
BIAS 0 



FEIGE- 
WATTS 
F 



CATEGORY 


IV 






<• 








ID2* 




100 


.558 


.0739 


.029 


.004 


.160 


IDl 


• 


10 


.442- 


.1831 


-.087 


.075 


.228 


CATEGORY 


III 




■ 






• 




Aai2 




10 


.531 


.0615 


.002 


.002 


.049 • 


DAD TXTr* 




10 


ceo 


. 1314 


• 029 


TOO 

. 129 


ACT 

.051 


tl C 1511 V C» 




c 
D 


.571 


. 0915 


.043 




ICO 

,252 


CLIMP • , 






TIT 

. 717 


, 3971 


1 O "7 

. 187 


. 016 


.230 


CATEGORY 


1 














SRAA2 




c 

5 


1 o c o 

1. o53 


. 0631 


1.324 


1. 295 


5/1.662** 


HSMATH, 




5 


.414 


.0248 


-.115 


-.100 


27.823** 


SAT2 




13 


.671 


.0670 


.142 


.150 


14.517** 


HSGPA2 




23 


.702 


.0287 


.173 


.150 


52.260** 


POPED 




6 


.911 


.1626 


.382 


.440 


5.635 


REPGPA 




7 


.917 


.0617 


.388 


.360 


53.256** 


NOBOOK 




5 


1.334 


.1133 


.805 


.800 


51.760** 


COLEFF 




4 


l./«6I 


.1160 


.932 


.765 


49..319** 


ANTDEG 




5 


1.631 


.2680 


1.102 


1.117, 


16.280*-. 


QCJOB 




4 


1.853 


.3533 


1.32A 


1.188 


14.227* 


PARASP 




5 


1.946 


. 7339 


1.417 


1.519 

i 


3.752 


Estimates from 


ingrouped data: 


b^jj = .529; 


SE(b,^) = 


.0032. 





Observed Bias = A = - 
*^Predicted Bias = 0 = Sy^.x^xz ~ ^ 

*^Feige-Watts F = A^(SS(A)) 



SS(fes)/in-l 
* Exceeds the 95 percent critical value for 
** Exceeds the 99 percent critical value for 
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Table A. 5 Es^timates from grouped data oi the standardized coefficients 
from the regression of ACH on SAT and bias prediction usin^ 
Hannan-Bur stein and Feige-Watts procedures. 



VARIABLE 
NAME 


NUMBER OF 
GROUP.S 
(n) 


B-^ 


SE(B--)^ 


OBSERVED^ l^J^" 
^ BURSTEIN 

PREDICTED 

BIAS 0 


FEIGE-^ 
WATTS 
F 


CATEGORY' IV 


\ 

\ 










ID2 


100 \ 


\832 


.0059 


-.007 


.003 


.015 


IDl 


10 


1.053 


.2168 


• 21i\ 


.029 


2.469 


CATEGORY III 














SAT2 ' 




.838 


.0190 


-.UUi 


— • UUJL 


.000 


PARING 


10 


.81? 


.0598 


-.022 


.021 


.143 


CLIMP 


4 


.576. 


.0388 


■:;.036 


.042 


.904 


POPED 


6 


.877^ 


.0685 


.039 


.038 


.331 


SRAA2 


5 


.899 


.0543 


.060 


.007 


.177 \ 


QCJOB 


A 


919 


.0261 


.073 


.C54 


7.982 


PARASP 


5 


.7.44 


.0903 


-.095 


-.059 


1.118 


NOBOOK 


5 


.718 


.0372 


-.121 


-.174 


11.096** 


CATEGORY I 














ACH2 


10 


1.168 


.0541 


.329 


.329 


197.730** 

f 


r>T7T> 015 A 

RLrbrA 


7 


1 . 019 


DAI fl 


.180 


.176 




COLEFF 


4 


1.054 


.;.i69 


.214 


.241 


3.438 


HSGPA 2 


23 


1.057 


.0329 


.218 


.219 


61.750** 


ANTDEG 


5 


1.120 


.0607 


.281 


.271 


21.759** 


HSPHYS 


5 


1.237 


.0422 


.398 


.295 


P8.739** 


HSMATH 


5 


1.396 


.0478 


.557 


.531 


149.284** 


Estimates from ungrouped 


data : 




SE(b^j^) = 


.0105 





ERIC 



Observed Bias = A = - 

^Predicted Bias = G = 8„_ ( 1 - ) 

I/, ./i A/. 2 

^ " 2 

^Feige-Watts F = A (SS(A)) 



ss(rs^)/m-i 



* Exceeds the 95 percent critical value f.or F* 
** Exceeds the 99 percent critical *yalue for F. 
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APPENDIX B: Estimating^ individual-level relations from grouped data 
when some data on individuals has been collected anony- 
mously* 

m 

The data^S^scribed in Appendix A are also used for the examples 

in this appendix. However, here we assume that information on one of 

the primary variables is collected anonymously while information on 

the other is identifiable (Context (D)). We are currently investigating 

various possibilities for circumventing this lack of information through 

aggregation techniques. Some preliminary findings are described below. 

Predicted Bias when a___. is Unknown — ^* n^i^ c i e 
XY The formula for predicting 

' ' ft - 

bias due to grbuping when the variables are statidardized at the individ- 
ual level is 

2 

where 6«„.v standardized regression coefficients and o- 

is the between group variance of the regressor. . 

^ In order to estimate Q we need' to know o ^ the covariance 

Y Z * A . X 1 

between regressor and regressor. But when data either X or Y has been 
collected anonymously and X and Y scores cannot be matched at the Individ 
ual level, we are unable to estimate 0^^^. Thus we are unable to estimate 



9 in this context 

A promising alternative 'Is to utilize the information conveyed . 
by the standardized coefficients from the regressions of Y on Z (By^^ 
and X and Z (B^^)^ to identify good grouping methods. If 6^^ ^ ^YZ*X* 
then an up*per bound for the predicted bias is given by 



0-1 



2 

c 



2 



In Tables B,l and B,2 information from Tables A. 4 and A,5 has 
been reproduced with the following modifications: 

(a) the Feige-Watts values have been deleted and 

(b) 7T values baaed on the information contained in 
Tables A, 2 and A, 3 have been included. 



Tables B,l and B,2 here 



The categorization scheme introduced by Burstein (1974a) is not\ 
as useful for distinguishing among grouping variables with high and low 
predicted tt's as it was with the predicted 9's, The estimates of tt from 
grouping variables tend to be inflated relative to the co ^sponding 
estimate of e when (5^2 small and \^>^^^2. ^^'S'' NOBOOK and POPED 
in the regression of ACH on SAT) • tt estimates are depressed relative 
to the corresponding 0*s when B^^^ \z moderate to large 

in magnitude and Sy2^Bj^2 HSGPA2 in the regression of ACH on SAT), 

Thus the rank ordering of grouping variables according to their 
e and TT values will differ. However, the problem of identifying good 
grouping methods is not completely hampered^ by this inconsistency if 
the investigator utilizes other common sense guidelines. 

First, he should group only variables for which ^^xz' 
This constraint would again eliminate most Category I 
variables. 

Second, any grouping variable should be eliminated for 
which B-- falls outside the bo^imds of po<ssJble values 
of Byj^* this case, grouping variables which yield 
values of B-- greater thanl should be eliminated since 
we are estimating a standardized coefficient. 



Table B.l Estimates from grouped data of the standardized 
coefficients- from the regression of SRAA on ACH, 
and predicted bias based on 9 and tt. 



VARIABLE 
NAME 


NUMBER OF 
GROUPS 
(m) 


YX 


SE(B")^ 
^ YX 


OBSERVED 
BIAS A 


PREDICT BIAS 
USING 

9 11 


CATEGORY IV 














ID?. 


100 


.558 


.0739 


.029 


.004 


.040 


IDl 


10 


A AO 


• XO J JL 


- 087 

< 


. U/D 




CATEGORY III 














ACH2 


10 


.531 


.0615 


.002 


.002 


.042 


PARING 


10 


.558 


.1314 


.029 


.129' 


.295 


HSPHYS 


5 


.571 


.0915 


.043 


.095 


.433 


GLIMP 


4 


.717 


. 3971 


..187 


.016 


.401 


CATEGORY I 














SRAA2 


5 


1.853 


.0631 


1.324 


1.395 


1.507 


HSMATH 


5 






- n s 


-.100 


.307 


SAT2 


13 


C.T\ 

. D/l 


nam 


• JLS ^ 


.150 


.210 


HSGPA2 


23 


.702 


.0287 


■ .173 


.150 


.451 


POPED 


6 


.911 


.1626 


.382 


.440 


.874 


REPGPA 


7 


.917 


.0617 


.388 


.360 


.635 


NOBOOK 


5 


1.33A 


.1133 


.805 


.800 


1.285 


GOLEFF 


4 


1.A61 


.1160 


.932 


.765 


i.194 


ANTDEG 


5 


1.631 


.2680 


1.102 


1.117 


1.586 
1.'630 


QGJOB 


4 


1.853 


.3533 


1.324 


1.188 


PARASP 


5 


1.946 


.7339 


1.417 


1.519 


2.048 


^Estimates from ungrouped data: 


Nx " 


.529; SE(b 


Y^) = .0032. 




^Observed Bias 


- a = - 













^Predicted Bias = 8 = 6„2.y6j^2 I ^ 

0^2 

''-=^Yz^xz = 

^ ;; 2 ' ^YZ-X 

''x 



' -I' 



Table B»2 Estimates from grouped data of the standardized 
coefficients from the regression of ACH on SAT 
and bias prediction based on 6 and it.- ' / 



/ 



VARIABLE 
NA>E 


NUMBER OF 
(m) . 




SE(B--)^ 


OBSER\ 
BIAS/ 

/ 


/ PREDICT BIAS 
USING 


CATEGORY TV 




- 

• 832/ 




/ 

/ 






ID2 


100 


.0059 


-.007 


.003 


.004 




10 


1.0^ 

/ 


.2168 ' 


.214 


.029 


. ^06; 


CATEGORY III 




/ 

i 








< 


SAT2 


1 1 

i J 


/ o o o 

/ . oJo 


. 0190 


-.001 


-.001 


.020 

( 


• PARING 


10 ^ 


/ 

.817 


.0598 


-.022 

V 


.021 


.252 


CLIMP 


A 
•t 


• O /O 


no Q o 


.036 


1 

.042 


.68(5 


POPED , 


6 


.877 


.0685 


.039 


.038 


.751 


SRAA2 


5 


. 899. 


.0543 


.060 


.007 


.067 


QCJOB 


4 


.912 


.0261 


.073 


.054 


.818 


PARASP 


5 


.744 


. 0903 


-.095 


-.059 


.5^3 


NOBOOK 


5 


.718 


.0372 


-.121 


17/. 


1.015 


CATEGORY I 














AGH2 


10 


1.168 


.0541 


.329 


.329 


.352 


REPCPA 


7 


1.019 


.0418 


.180 


.176 


.695 


GOLEFF 


4 


1.054 


.1169 


.214 


.241 


.828 


-HSGPA2 


23 


1.057 


.0329 


.218 ■ 


.219 


.095 


ANTDEG 


5 


1.120 


,»0607 


.281 


.271 


•■ .751 


HSPHYS 


5 


1.237 


.0422 


.398 


.295 


.858 


HSMATH 


5 


1.396 


.0478 


.557- 


.531 


1.189 



Estimates from ungrouped data; b = .529; SE(h ) = 

YX YX 

Observed Bias = A = B-- - h^^ 



.0032, 



-Predicted Bias = 6 ^g^^.^g^^ 



( 1 - ^x-^) 
%2 



^z^xz (—"2) = hz^ e 



a«2 

t. 



8. 
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Kj,n/illy, grouping \^ariables for which the predicted bias 
Ik very large can be dropped from further consideration. 

This last guideline is the most controversial and aiabiguous 
as it is not obvious what a very large tt" will be in any particular 
study. Clearly, the predicted tt (1.016) for NOBOOK in the ACH-on-SAT 
regression is very large, but are the tt^s for POPED (.751) and CLIMP 
(.686) also very large? Not necessarily; it all depends on the magni- 
tude of the estimates of g^x- ^^^"^ other grouping variables and on the 
magnitude of tt for other potential Z/s. 

In any case, it is reassuripg that in both tables, the variables 
that have lower tt values and also meet th^ three common sense guidelines 
provide reasonably accurate estimates of B^^* next section we 

describe an approach that combines' variables with smallest predicted 
bias and thereby affords greater confidence in estimates^-f rom grouped 
data. 

Composite Estimates from Multiple Grouping Variables . The above findings 
suggest that even in Contexf (D), an 'investigator can distinguish those 



to rea 
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grouping characteristics which lead to reasonably accurate estimates 
from those providing extremaly misleading ones. Once this separation 
has been accomplished, the investigator can choose the characteristic 
with the smallest predicted bias. ^Better yet, he can use the available 
information about each characteristic and its expected bias to form a 
weighted composite of good grouped estimates. For example, grouped es- 
timates can be weighted in an inverse proportion to their predicted bias. 
The standard errors of the grouped estimates or the number of groups formed 
(m) can also be used to give additional weight to the potentially more 
stable estimates . in Context (D) where a^^ is unknown, we cannot group 
on the regresscr, and we are trying to generate estimates that are better 
than those from random grouping (Category IV), Our compositing procedure 



works- as follows: 

(1) Identify the 3 grouping variables (other than the rep,ressor 
or Category IV variables) with smallest ^ values, excluding 
those variables for which 1 1.0. ' . ' 

(2) Find the Fisher Z-tr:lnsforTnation for the B~ of each of the 
3 variables. 

(3) Weight the Fisher-Z values: 

(a) in inverse proportion to the predicted biases 

3 

7T* = ( E 7T^ - TT^) where i identifies a specific grouping 
i=l variable; 

(b) by the number of groups formed (m) ; 

(cx) in inverse proportion to the squared standard error of 
the grouped estimate: 

(SE(B--))«* =(J^(SE(B--)\)- (SE(B^-))2 

(4) Find the weighted average of Fisher-Z values in each case: 

(5) Transform the average Fisher-Z back to correlation units 
to find the composite estimate of B^^. 

'\ ' 

There are other possible weighting methods and any number of weight- 
ing methods can be combined to generate a new weighting scheme. The ones 
included here are intended only to illustrate the technique. 

In Table B.3, we present composite estimates for the standardized 
coefficient from 3 regressions ~ (A) SRAA on ACH, (B) ACH on SAT, (C) SRAA 
on SAT-. Information, on the grouping variables that contribute to the com-, 
posites is also provided. 



Table B.3 here 



The results of the compositing process are satisfactory. In each 
example at least one of the composite estimates is more accurate than any 
Category III or Category IV variable with the exception of grouping on 
the regressor. The practical utility of the composite estimates is high 



Table B,3 Weighted composites from grouped estimates, of 
standardized regression coefficients 



A. Regression of SRAA on ACH ~ b^^^ « .529 

Grouping No. of SE(B") Predicted 

Variable Groups (m) YX ^ YX g^^^ 

PARING 10^ -558 .1314 .295 

HSMATH 5 .414 ,0248 .307 

SAT2 13 .671 ' .0670 .210 

Estimates from Weighted composites 
Weights 

Ogtermined by - Estimate 

If* .564 

m' .594 
(SE(B~))* .545 



B. Regression of ACH on SAT — b^j^ = .839 



Grouping ■ No. of sFfB— ■) Predicted 

Variable Groups (m) YX ^ YX\ . Bias (tt) 

PARING 10 .817 .0598 .252 

SRAA2 5 .899" .0543 .067 

PARASP 5 .744 .0903 .533 

Estimates from Weighted composites 
Weights 

Determined by Estimate 
* .852 

TT 



.828 



(SE(B~))^* .844 



Table B»3 (C) cont* 





C. Regression of SRAA on SAT 


- b^j, - .574 


Grouping 
Variable 


No. of 

Groups YX 


SE(B~) 

I A 


r reaic ted 
Bias (tt) 


ACH2 


10 .651 


.0750 


.575 


CLIbff 


4 .672' 


..3960 


.637 


PARING " 


lo' .434 


.1316 


.237 


Estimates from Weighted composites 






Weights 
Determined by 


Estimate 






ir* 


/ .576 






m 


/ .574 




• 


(SECB^jj))^* 


/ .564 







/ 
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Jn every case 5^Lnfce the wor»t of the composite estimates (•594, based 
on the SRAA-on-ACH Y^gression where h^^ = ,529) deviates by only 12% 



from the ungrouped value. This implies a ,12 standard deviation over 
prediction from this compositing procedure. 

Much remains to be done before we can prescribe uniformly powerful 
methods of describing how many variables to include in flie composite and what 
method should be used to weight the estimates. However, it is obvious ttet the 
compositing process has merit especially when more direct approaches to choo 
ing the best grouping variable are not possible as in Context (D) • 
Reconstructing a Correlation Matrix from Grouped Data , We are also 
^xploring the feasibility of accurately reconstructing individual-level 
correlation matrices from estimates based on grouped observations. The 
procedures we have examined require the standardization of all primary . 
variables before grouping, and knowledge of the zero-order correlation 
coefficients relating each grouping variable (Z) to the primary variables 
(here designated X^^), These correlations are presented in Table B,4, 



^ Table B,4 here 



For any two primary variables and X2> and each' grouping variable Z, we 

regressed X on X (weighted group means) when r is moderate to large 

JL z 2 

and r " > r , When r is moderate to large and r J _ > r , we re- 

2 1 1 1 2 

gressed X and X , This procedure yields a pool of v Bv v values 

Z 1 X • 'X^X2 X^X^ 

that can be used to estimate r , 

^1^2 

By examining the estimates generated in the above fashion, we were 
successful in reconstructing a correlation mattix. In Table B,5, the 
individual-level correlations coefficients among five primary variables 
are presented in the upper triangular portion of the matrix. The best 
estimates from grouping on a variable other than the regressor or random 

00049 
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Table B.4 Zero-Order correlations relating potential 
grouping variables to primary Variables. 



CORRELATION COEFFICIENTS 



Grouping 
Variable 


SRAA 


Primary Variables 
ACH SAT 


HSGPA \ 


H<?PtTV<? 


ANTDEG 


.264 


.156 - 


.140 


.046 




CLIMP 


.074 


.147 ■ 


.165 




\ mi 


COLEFF 


.189 


.134 


.114 




\ non 


HSMATH 


. 202 


.479 


.346 


,248 


.358 

• i ■ 


HSPHYS- 


• .209 


.318 


.257 


.132 




NOBOOK 


.196 


.146 


.203 


:030 


.021 


PARASP 


.172 


.066 


.087 


.001 


.109 


PARINC 


.064 


.070 


.076 


-.101 


-.009 


POPED . 


.145 


.139 


.157 


-.010 


.007 


QCJOB 


.199 


.106 


.118 


.087 


.040 


REPGPA 


-.455 , 


-.490 


-.468 


-.810 


-.108 
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grouping (to mirror our usual state of knowledge in Context (D)), are . 
contained in the lower triangular region. All values are repeated to 
two digits only. 

Table B.5 here 

The fit is remarkable given the coarseness of the grouping methods in 
our example. Only one grouped estimate deviates .by more than .02 from 
its corresponding ungrouped correlation coefficient. A statistical test 
of Goodness of fit would be superfluous even with a sample size of 2676. > 
The above example reflects the potential rather than the present. *^ 
Firm guid-elines for choosing the best grouped estimates for reconstruct- 
ing a correlation matrix have not yet been developed. Here we knew the 
values needed and this enabled us to pick and choose among potential 
grouping methods. In practice the investigator with anonymously collected 
information on some primary variable is not so fortunate. 
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Table B.5 Individual-level correlation matrix reconstructed from 
standardized between-group regression coefficients • 



Individual-Level Coefficients (Upper Trianr,ular Region ) 





VARIABLES 


SRAA 


ACH 


SAT 


HSGPA 


HSPHYS 




SRAA 




.53 


.57 


.37 


.21 


Best 


ACH 


.55 




.84 


.53 


.32 


Estimates 














From Grouped 


SAT 


.58 


.82 




.49 ■ 


.26 


Da ta (Lower 












Triangular 


HSGPA 


.36 


.52 


.54 




.13 


Region) 
















HSPHYS 


.21 


.34 


.27 


.13 ' 






APPKrinTX » FOOTNOTKS 



1 . ^ , 

Since all variables are standardized, this is equivalent to examinig 

the zero-order correlation coefficients p^^ Pj^2* fact, even 

when variables are unstandardized^ it is better to compare correlations 
as it is the relative strength of relation that is important without 
regard to differences in variation of the variable. 

2 

It is. possible to specify the conditions under which ^ ^yZ*X ^^^^^ 
X, Y, and Z are standardized. 

»■ 

^YZ " ^Z-X 

when ^ ' ' 

-Byz^YZ-X ^YX ^X7 positive, or when 

or when 

(ii) Byz^XZ ^ ^YX ^XZ negative. 

In most cases, researchers will have some guidance as to whether 
jrhese conditions hold even when data are collected anonymously^ 
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ERRATA 



Data Aggregation in Educational Research:' Applications 
Page 1~ 

Mayeske et al* should be dated 1973 . KLso only one of tlie volumes is 
cited anong the references.' 

The ISA studies are included in the references under the names of the 
authors (Husen/ Cjcmber and Keeves, Purves, Peaker, and Thomdike) . Not all 
of the volim^s are listed as sane are still being revised and are not discussed 
in the paper. The correct publication dates are 1967 for the Husen volxxGfK^, 
197 3'^ for the American edition of Comber and Keeves, Purves, and Thomdike, anu 
hopefully, 1975 for Peaker. 

Page 5 — 

The correct date for the Blalock reference is 1972 ( Social Forces) . 

The correct date for the Wiley and Wiley is 1970 ( American Sociological Review) < 

'The first sentence of the last paragraph should read: 

"Sound principles have already been developed. . . " ("been" left out of earliei 
draft.) , ' . 

Page 6 — 

")" left off end of set of references in- first 2 lines of page. 

Page 8 — ^ . 

Delete after Scheuch*s name in 2nd line fron bottom. 

Page 9 — 

- The correct publication date for the Katzman; reference is 1968. 
")" after Levin (1970) in last paragraph. 

Page 10— 

The Summers and ^felfe (1974) reference for the, Federal Reserve Bank of 
Philadelphia is in error. There are actually two reports from there so far. 
One appeared in Business Review (Summers and WoXfe) and the other v^s a presentatior 
to the Econcmetric Society (presumably Surmiers) . The ccnments in the paper on 
the study are based on discussions with Summ^s and a suimary of the Econcmetric 
Society paper. 

Page 12— 

The last footnote to Table 2 ("f'**) reads " Ten year-olds were not included 
in the Literature study." 

Page 14 — 

The heading " Individual Model" after "1." should read " Individual Level ". 

Page 16 — , I 

The Heading stould read "CHOOSING "THE APPRDPKEATE UNIT OF ANALYSIS" (singular) ^ 

Lines 5 and 6 after heading— sentence is missing "of" , should read | 
"...the specification of a set of research foci..." 1 

Page 17-- 

Sane versions are missing the reference to the Rand sponsored study by 
Klitgaard and Hall (1973) . 

Page 19— 

The study of administrative intensity cited on the 6th line is by Freeman 
and Hannan (1975) (Authorship position reversed) . 
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Page 19 (cont. ) 

There has been sorve modification of the paragraph about the Walberg and 
Rasher (1974) study. Unfortunately , the modifications were made too late 
for inclusion in the duplicated version of the paper. The paragraph should 
read: ' * ^ 

" For example, .though their conclusions are appealing, Walberg and 
Rasher (1974) cannot avoid the methodological shortcomings of the Coleman 
study by using state- level data, especially with 1969 and 1970 selective 
service examination failures as the outcane ^variables. Walberg had 
previously cited Robinson's (1950) paper ( Walberg, 1969, p. 530) and 
thus should be aware of the danger of treating states as randan groupings 
of persons. Besides, jia) women did not take military selective mental tests 
and (b) the years 1969-70 \>?ere not the times to try to pass the test 
( a case of the winners are the losers, in this writer's opinion). It is - 
particularly unfortunate that this article appeared in a journal that is 
more widely read by administrators than by researchers since the former are 
less likely to realize its methodological liod.tations," I 

Page 20 — * ^ 

Glass and Stanley's book has a publication date of 1970 rather than 1972. 

.Page 21— ' 

Tae correct references to Hauser are 1969, 1970, 1971, 1974 • 

Page 22— 

The subheading for the Lewy study should read "Group Anchored versus Sample 
Anchored if4easures". 

FOOTi\XDTES-- 

Footnote 2 — Citation should read See, e.g., Hauser (1974). 
References — 

The following citations have incorrect publication dates in earlier versions: 

Ccxnber and Keeves (1973) 
Peaker (1975) 
Purves (1973) 
Thomdike (1973) 

Also, there are unnecessary parentheses around the citations for Boruch (1972b) 
and Burstein (1974a) . 
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